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DEFINITIONS 

IDA  publishes  the  following  documents  to  report  the  results  of  its  work. 


Reports 

Reports  are  the  most  authoritative  and  most  carefully  considered  products  IDA  publishes. 
They  normally  embody  results  of  major  projects  which  (a)  have  a  direct  bearing  on 
decisions  affecting  major  programs,  (b)  address  Issues  of  significant  concern  to  the 
Executive  Branch,  the  Congress  and/or  the  public,  or  (c)  address  issues  that  have 
significant  economic  implications.  IDA  Reports  are  reviewed  by  outside  panels  of  experts 
to  ensure  their  high  quality  and  relevance  to  the  problems  studied,  and  they  are  released 
by  the  President  of  IDA. 

Group  Reports 

Group  Reports  record  the  findings  and  results  of  IDA  established  working  groups  and 
panels  composed  of  senior  individuals  addressing  major  issues  which  otherwise  would  be 
the  subject  of  an  IDA  Report.  IDA  Group  Reports  are  reviewed  by  the  senior  individuals 
responsible  for  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
relevance  to  the  problems  studied,  and  are  released  by  the  President  ol  IDA. 

Papers 

Papers,  also  authoritative  and  carefully  considered  products  of  IDA,  address  studies  that 
are  narrower  in  scope  than  those  covered  in  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  expected  of  refereed  papers  in  professional  journals  or 
formal  Agency  reports. 

Documents 

IDA  Documents  are  used  for  the  convenience  of  the  sponsors  or  the  analysts  (a)  to  record 
substantive  work  done  in  quick  reaction  studies,  (b)  to  record  the  proceedings  of 
conferences  and  meetings,  (c)  to  make  available  preliminary  and  tentative  results  of 
analyses,  (d)  to  record  data  developed  in  the  course  of  an  investigation,  or  (e)  to  forward 
information  that  is  essentially  unanalyzed  and  unevaiuated.  The  review  of  IDA  Documents 
is  suited  to  their  content  and  Intended  use. 
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PREFACE 


The  issue  analysis  and  resolution  process  for  Ada  compiler  validation  was 
requested  by  the  Ada  Joint  Program  Office  (AJPO),  under  the  Institute  for  Defense 
Analyses  (IDA)  Task,  Ada  Validation,  as  an  on-going  part  of  the  Department  of  Defense’s 
Ada  language  control  program.  The  issues  resolved  during  fiscal  year  1992  are  the  subject 
of  this  report  which  is  a  partial  fulfillment  of  this  task. 

Special  thanks  are  due  to  Ms.  Christine  Anderson,  Ada  9X  Project  Manager,  for  her 
efforts  to  bring  the  economic  and  procedural  concerns  of  compiler  vendors  to  the  attention 
of  the  Ada  Board  and  participants  in  the  Ada  validation  process.  Also,  thanks  are  due  to 
members  of  the  Fast  Reaction  Team  who  conscientiously  gave  of  their  time  to  assist  IDA 
with  the  analyses  and  resolution  of  complex  language  and  compiler  implementation  issues: 
Dr.  Robert  Dewar  (New  York  University),  Dr.  John  B.  Goodenough  (Software  Engineering 
Institute),  Dr.  Norman  Cohen  (IBM),  Dr.  Stephan  Heilbrunner  (Salzburg  University),  Dr. 
Erhard  Ploedereder  (Tartan  Laboratories),  Dr.  Brian  Wichmann  (National  Physical 
Laboratory-UK),  and  Dr.  Kenneth  Dritz  (Argonne  Laboratories).  The  IDA  reviewers  of  this 
report  were  Dr.  Dennis  W.  Fife,  Dr.  Cy  D.  Ardoin,  and  Dr.  Richard  L.  Wexelblat. 
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EXECUTIVE  SUMMARY 


The  Ada  Joint  Program  Office  (AJPO)  has  tasked  the  Institute  for  Defense  Analyses 
(IDA)  to  conduct  on-going  analyses  of  the  Ada  compiler  validation  policies,  procedures, 
and  conformity  test  programs.  During  fiscal  year  1992,  validation  policies  were  the  primary 
area  of  concern  among  compiler  vendors  anticipating  the  requirement  for  significant  capital 
investment  in  Ada  9X  compilers.  Several  proposals  for  policy  change  were  made  by  the 
Ada  Board,  a  Federal  Advisory  group,  who  advise  the  AJPO  Director  on  a  wide  range  of 
Ada  Program  topics.  IDA  participated  in  the  process  of  resolving  issues  raised  by  Ada 
Board  proposals  and  other  issues  raised  by  DoD  and  non-DoD  organizations.  IDA  analyzed 
these  proposals  for  technical  and  procedural  implications.  Recommendations  for  the 
Director,  AJPO,  were  based  upon  IDA’s  analysis  and  comment  from  various  individual 
experts  and  organizations  having  an  interest  in  conformity  testing.  This  report  summarizes 
the  issues  that  had  a  bearing  on  policy  changes  incorporated  in  a  revision  of  the  Ada 
Compiler  Validation  Procedures  produced  by  IDA  and  published  by  the  AJPO  in  August 
1992.  These  policy  changes  include: 

•  A  time-limited  (one  year)  provisional  validation  status  for  a  compiler  vendor 
who  fails  a  small  number  (ten  or  fewer)  conformity  tests. 

•  Designation  of  the  Ada  Compiler  Validation  Capability  (ACVC)  test  suite 
version  1.11  as  the  final  version  for  testing  compilers  that  implement  the  1983 
version  of  the  Ada  language. 

•  Guidelines  for  DoD  software  project  managers  on  configuration  management  of 
an  Ada  compiler  over  the  life  of  a  software  development  project. 

IDA  also  maintains  a  current  and  historical  data  base  containing  pertinent 
information  about  technical  issues  raised  by  compiler  vendors  who  are  using  the  test 
programs  contained  in  the  ACVC.  This  report  also  summarizes  the  disputed  tests  and  the 
rationale  for  resolution  of  them  that  was  entered  into  this  data  base  during  fiscal  year  1992. 
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1.  INTRODUCTION 


The  Ada  Joint  Program  Office  (AJPO)  has  tasked  the  Institute  for  Defense  Analyses 
(IDA)  to  provide  independent  analyses  of  the  Ada  compiler  validation  process  and  to  assist 
in  resolving  technical  and  procedural  issues  that  arise  during  the  execution  of  the  process. 
IDA  provides  a  range  of  technical  and  policy  analyses  to  support  the  AJPO,  primarily  act¬ 
ing  as  an  arbiter  of  issues  that  could  reduce  the  effectiveness  of  DoD’s  policy  for  the  use  of 
validated  Ada  compilers.  Many  interested  parties  provide  comments  on  Ada  compiler  val¬ 
idation  policies.  IDA  assists  the  AJPO  in  evaluating  these  proposals  for  potential  policy 
change  and  in  developing  implementation  procedures  when  a  policy  decision  has  been 
made.  This  report  discusses  the  validation  policy  changes  that  have  been  incorporated  in 
the  Ada  Validation  Procedures,  Version  3.1.  There  is  also  a  summary  of  the  test  issues 
resolved  through  the  Ada  Compiler  Validation  Capability  (ACVC)  test  dispute  process. 

This  fiscal  year  summary  of  work  performed  on  behalf  of  the  AJPO  was  written  for 
those  who  have  an  interest  in  recent  validation  issues  and  for  those  who  will  consider  var¬ 
ious  policy  proposals  to  align  the  current  process  with  the  requirements  of  the  revised  lan¬ 
guage  and  test  suite  known  as  Ada  9X. 


2.  BACKGROUND 


During  1991  and  1992,  compiler  validation  issues  were  raised  by  the  Ada  Board, 
the  Services,  and  compiler  vendors  that  resulted  in  policy  changes  in  the  current  validation 
process.  The  resolution  of  issues  concerning  the  present  practices  was  decided  by  the  Direc¬ 
tor,  AJPO,  and  incorporated  in  a  revised  Ada  Validation  Procedures  document  published  in 
August  1992.  These  policy  changes  generally  provide  economic  benefits  for  compiler  ven¬ 
dors  in  anticipation  of  their  being  able  to  invest  resources  in  tools,  performance  improve¬ 
ments,  and  Ada  9X  implementations. 


3.  POLICY  ISSUES 


This  section  summarizes  the  validation  policy  issues  and  actions  taken  to  resolve 
them.  The  resolution  of  some  issues  concerning  Ada  9X  is  still  in  progress  as  part  of  the 
Ada  9X  project. 

3.1  PROVISIONAL  VALIDATION  STATUS 

Issue:  Validation  costs  can  be  reduced  by  changing  the  test  dispute  process 
and  allowing  a  compiler  to  fail  a  small  number  of  tests  in  the  ACVC  test 
suite. 

Several  Ada  Board  members  raised  the  issue  of  the  cost  borne  by  compiler  vendors 
to  dispute  the  correctness  of  tests  in  the  ACVC.  Because  vendors  are  uncertain  of  the  out¬ 
come  of  a  dispute  and  how  long  it  will  take  to  get  a  resolution,  some  Ada  Board  members 
claimed  that  vendors  do  not  bother  to  dispute  tests  but  change  their  compilers  to  pass  them. 
Often  making  these  compiler  changes  diverts  vendor  resources  from  improvements  that 
would  benefit  the  user  community. 

The  Ada  9X  effort  and  cost  to  vendors  to  implement  Ada  9X  compilers  became 
inextricably  related  to  the  problems  of  ACVC  quality  and  the  requirement  that  a  compiler 
must  pass  all  applicable  ACV C  tests  as  a  condition  for  receiving  a  validation  certificate. 
Several  members  of  the  Ada  Board  prepared  a  position  paper  for  the  Director,  AJPO,  in 
which  they  recommended  that  a  vendor  need  not  pass  all  ACVC  tests  to  obtain  a  certificate 
because  some  tests  are  known  to  have  marginal  value.  It  was  proposed  that  a  vendor  could 
dispute  any  test  in  the  ACVC  as  being  incorrect  or  pathological  (i.e.,  the  latter  being  a  test 
for  a  language  rule  that  is  not  useful  in  practice).  Test  disputes  would  be  resolved  by  an 
extended  process  of  deliberation  by  the  International  Organization  for  Standards  Working 
Group  on  Ada  (ISO  WG-9)  who  would  issue  a  resolution  in  due  time  without  the  pressure 
to  adhere  to  an  Ada  Validation  Facility  (AVF)  contract  schedule  for  validation.  The  Ada 
Board  paper  suggested  that  IDA’s  role  should  be  limited  to  withdrawing  ACVC  tests  that 
contain  obvious  errors  and  resolving  test  issues  based  upon  precedent  for  resolutions.  The 
arguments  for  an  extended  test  resolution  period  were  based  on  the  view  that  a  vendor 
should  be  allowed  to  have  a  certificate,  even  though  some  ACVC  tests  were  failed,  while 


ISO  WG-9  considers  the  merits  of  the  dispute  more  thoroughly.  The  extended  resolution 
period  would  not  be  constrained  by  a  target  date  for  issuing  a  resolution  as  is  the  case  in  the 
present  resolution  process.  At  the  end  of  the  ISO  WG-9  deliberation  period,  if  the  test  was 
judged  to  be  both  correct  and  useful,  the  vendor  would  be  obligated  to  pass  the  test  in  future 
validations  of  that  compiler. 

Several  benefits  were  expected  from  the  extended  resolution  process.  From  an 
ACV C  quality  viewpoint,  vendors  would  be  assisting  in  the  discovery  of  pathological  tests. 
ISO-WG-9  could  directly  assist  the  Ada9X  ACVC  Team  to  avoid  perpetuating  similar  tests 
in  a  new  test  suite.  Finally,  vendors  could  forego  the  cost  of  a  dispute  before  obtaining  a 
validation  certificate  and  could  use  these  resources  for  work  on  tools  and  Ada  9X  compil¬ 
ers. 


The  Ada  Board  paper  was  distributed  to  AVF  managers  and  IDA  prepared  a  draft 
revision  of  the  Ada  Validation  Procedures.  The  AJPO  provided  the  DoD  Senior  Software 
Officials  with  these  revised  procedures  for  review  and  comment.  Service  and  Agency  com¬ 
ments  provided  little  support  for  the  Ada  Board  recommendations  with  vigorous  objections 
coming  from  the  AVFs.  It  was  not  clear  to  the  reviewers  why  a  change  in  policy  was  need 
for  validating  Ada  83  compilers  (i.e.,  compilers  validated  under  the  1983  Ada  language 
standard)  and  others  objected  to  the  principle  of  issuing  a  validation  certificate  for  a  com¬ 
piler  with  known  errors.  Yet  the  problem  of  how  to  reduce  validation  cost  now  so  that  com¬ 
piler  vendors  could  invest  in  tools  and  Ada  9X  compilers  remained  an  issue  that  needed 
resolution. 

Resolution:  IDA  prepared  an  analysis  of  the  Ada  Board  paper  for  the  Director, 
AJPO  that  recommended  a  compromise  position.  The  Ada  Board’s  major  concerns  were 
providing  economic  relief  to  compiler  vendors  and  improving  the  quality  of  the  ACVC 
tests.  The  IDA  “Strawman”  proposal  recommended  the  following: 

•  A  compiler  vendor  should  be  able  to  complete  a  contracted  validation  process 
with  not  more  than  10  failed  ACVC  tests,  and  that  the  compiler  be  re-tested 
within  a  12-month  period,  and  successfully  pass  the  ACVC,  to  obtain  a  Valida¬ 
tion  Certificate. 

•  A  compiler  vendor  should  request,  at  any  time,  that  ISO  WG-9  accept  a  partic¬ 
ular  ACVC  test  for  extended  resolution.  IDA  will  withdraw  any  test  accepted 
for  extended  resolution  so  that  all  compiler  vendors  will  be  exempt  from  the  test 
until  a  final  resolution  has  been  made  by  ISO  WG-9. 
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♦  The  current  test  dispute  process  should  continue  with  the  only  modification 
being  that  a  vendor  need  not  have  all  disputes  resolved  before  commencing  the 
on-site  testing  process.  A  vendor  may  declare  the  failure  of  up  to  10  tests  and 
elect  to  obtain  a  provisional  validation  status  while  corrections  are  made  to  the 
compiler.  If  the  vendor  appeals  to  ISO  WG-9  for  the  extended  resolution  pro¬ 
cess,  a  validation  certificate  may  be  issued  if  all  the  failed  tests  are  accepted  by 
ISO  WG9.  IDA  will  also  withdraw  the  tests  accepted  for  the  extended  resolution 
process  so  that  no  other  vendor  need  dispute  them. 

•  If  a  compiler  vendor  does  not  successfully  pass  the  ACVC  within  12  months, 
the  provisional  validation  status  expires. 

The  Director,  AJPO,  accepted  the  Strawman  recommendations  with  an  added  pro¬ 
viso  that  a  certificate  of  provisional  validation  status  would  contain  the  list  of  failed  tests. 
A  list  of  compilers  that  have  provisional  validation  status  and  the  identification  of  failed 
tests  would  be  public  information.  Provisional  validated  status  is  a  major  policy  change  that 
should  encourage  compiler  vendors  to  respond  to  market  demand  for  new  Ada  compilers 
(e.g.,  new  hardware  architectures)  with  a  grace  period  to  appeal  failed  tests  to  ISO  WG-9 
or  to  change  the  compiler. 

3.2  CHANGE  IN  THE  TEST  SUITE  CONTENT 

Issue:  The  problem  of  validation  costs  is  a  continuing  concern  that  is  relat¬ 
ed  to  the  frequency  of  change  in  the  ACVC  and  the  number  of  errors  in  each 
version. 

The  conformity  test  suite  for  Ada  83  evolved  over  a  ten-year  period  with  new  ver¬ 
sions  being  issue  every  6  months,  at  first,  then  12  months,  and  finally,  every  18  months.  The 
life  of  a  Validation  Certificate  was  initially  12  months  and  was  increased  to  be  consistent 
with  the  18-month  ACVC  release  cycle.  Each  new  version  of  the  ACVC  has  contained  new 
or  changed  tests  which  contain  errors.  For  example,  in  ACVC  1.11  over  10%  of  the  new  or 
changed  tests  tests  were  withdrawn.  Vendors  claim  that  internal  cost  for  configuration  man¬ 
agement  of  the  test  suite,  pre-validation  testing  on  all  their  compilers,  and  the  research 
required  to  dispute  a  failed  test  far  exceeds  the  formal  validation  cost  of  AVF  testing.  After 
initiation  of  the  Ada  9X  Project,  vendors  expressed  concern  about  continuation  of  the  18- 
month  version  release  schedule.  The  release  schedule  was  extended  to  24  months  in  1990 
but  vendors  challenged  the  need  for  new  releases  of  a  test  suite  for  a  language  undergoing 
revision.  Each  new  release  would  contain  some  tests  with  errors  that  could  only  be  discov¬ 
ered  after  vendor  resources  had  been  expended  to  find  them.  The  consensus  among  vendors 
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and  users  was  that  adding  new  tests  would  not  only  provide  marginal  benefits  but  could 
conflict  with  some  Ada  9X  revision  goals. 

Resolution:  In  the  Spring  of  1992,  the  Director,  AJPO,  announced  to  the  Ada  com¬ 
munity  that  tests  for  validation  of  Ada  83  compilers  had  been  frozen  on  version  1.11.  Tests 
could  be  removed  but  no  tests  would  be  added  or  changed.  There  were  few  benefits,  except 
to  AVF  revenue,  by  issuing  a  changed  version  of  the  test  suite.  Most  of  the  incorrect  tests 
had  already  been  withdrawn  by  IDA,  with  documentation  having  been  provided  to  the  Ada 
9X  ACVC  team  and  the  Ada  9X  ACVC  reviewers. 

The  Ada  9X  Project  Office  is  committed  to  reduce  the  number  of  marginal  tests  by 
deriving  tests  that  reflect  the  use  of  the  Ada  language.  Although  there  is  no  objective  mea¬ 
sure  of  language  use,  the  Ada  9X  ACVC  Team  and  ACVC  Reviewers  are  investigating 
many  sources  of  information  concerning  language  use.  It  is  recognized  that  the  number  of 
tests  for  similar  objectives  could  be  reduced  by  combinations  of  test  code  and  analysis  of 
the  unique  test  cases  that  demonstrate  conformity  in  a  normative  context.  Exhaustive  test¬ 
ing  of  all  possible  contexts  is  impractical.  IDA  continues  to  support  the  Ada  9X  Project 
Office  in  resolving  the  issue  of  quality  and  cost  containment  for  the  Ada  9X  ACVC. 

3.3  UNDETECTED  COMPILER  ERRORS 

Issue:  All  validated  compilers  contain  undetected  errors.  The  conformity 

test  suite  should  be  concentrated  on  finding  errors  that  affect  the  portability 

of  programs  produced  by  the  compiler. 

The  resolution  of  this  issue  is  closely  related  to  finding  a  reasonable  language  use 
basis  for  selecting  ACVC  tests  but  is  more  narrowly  focused  on  deviations  among  compil¬ 
ers.  ISO  WG-9  does  provide  advisories  to  compiler  implementers  and  users  on  potential 
deviations  that  adversely  affect  the  portability  of  Ada  programs.  The  Ada  9X  language  will 
provide  guidance  for  the  use  of  the  language  in  certain  user  domains  such  as  real-time, 
embedded  systems,  business  systems,  and  safety  critical  applications.  The  Ada  9X  ACVC 
will  reflect  the  specialization  these  domains  represent. 

Resolution:  It  is  too  early  to  say  whether  the  work  of  the  Ada  9X  Project  and  ISO 
WG-9  will  provide  test  objectives  for  portability  within  specific  user  domains.  The  Ada  9X 
testing  strategy  will  be  different  in  some  respects.  Experience  with  the  test  suite  will  be 
needed  to  revise  the  Ada  Validation  Procedures  appropriately  and  to  arbitrate  the  test  dis¬ 
pute  process. 
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3.4 


PERIODIC  COMPILER  REPLACEMENT 


Issue:  It  is  impractical  for  DoD  software  project  managers  to  replace  an 
Ada  compiler  version  just  because  its  certificate  has  expired.  Project  man¬ 
agers  should  be  given  some  practical  guidance  about  how  to  comply  with 
the  Ada  mandate. 

Managers  of  software  development  projects  select  an  Ada  compiler  that  has  vali¬ 
dated  status  but  the  life  of  validated  status  is  tied  to  the  expiration  date  of  the  certificate. 
Compiler  vendors  offer  maintenance  agreements  that  include  replacement  of  the  compiler 
with  a  newer  version  having  a  certificate.  The  replacement  compiler  may  be  different  in 
some  respects  so  that  source  code  developed  using  the  earlier  version  must  be  recompiled. 
Software  managers  prefer  to  select  a  particular  version  of  a  compiler  for  use  during  an 
entire  development  period.  However,  when  the  developing  organization  is  ready  to  deliver 
the  software  to  a  maintenance  organization,  the  receiving  organization  may  refuse  to  accept 
the  software  because  the  compiler  does  not  have  validated  status. 

As  more  and  more  Ada  systems  are  being  turned  over  to  the  government  or  contrac¬ 
tors  for  maintenance,  DoD  project  managers  and  system  program  managers  have  petitioned 
the  AJPO  for  adjudication  of  claims  that  the  delivered  code  does  not  perform  as  expected 
because  it  was  developed  using  a  compiler  that  does  not  meet  the  literal  interpretation  of 
validated.The  Services  requested  that  the  Ada  Validation  Procedures  provide  guidance  to 
project  managers  on  the  use  of  the  Ada  compiler  validation  process.  Non-DoD  software 
development  organizations  objected  to  including  guidance  from  DoD  on  the  software 
development  practices  they  employ. 

Resolution:  An  appendix  was  added  to  the  revised  Ada  Compiler  Validation  Pro¬ 
cedures,  Version  3.1,  to  provide  guidance  to  DoD  project  managers.  The  guidance  is  based 
on  configuration  management  practices  to  control  and  document  changes  to  the  compiler 
and  re-testing  of  the  compiler  with  the  ACVC  version  used  to  validate  the  compiler.  The 
results  from  this  re-testing  indicate  the  differences  caused  by  compiler  changes  and  identi¬ 
fies  any  resulting  non-conformities.  A  project  manager  will  determine  what  action,  if  any, 
will  be  taken  to  bring  the  compiler  up  to  a  current  validated  version.  Maintenance  organi¬ 
zations  will  also  be  provided  with  complete  and  current  technical  documentation  of  the 
compiler.  Non-DoD  organizations  are  not  obligated  to  follow  this  guidance  but  may  choose 
to  incorporate  some  aspects  in  their  procedures. 
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4.  ACVC  TEST  DISPUTES 


In  fiscal  year  1992,  AVFs  submitted  over  40  petitions  against  ACVC  tests.  Most  of 
these  disputes  were  resolved  by  precedent  but  there  were  nine  disputes  that  had  not  been 
raised  previously.  Among  the  new  disputes,  the  principal  issues  addressed  in  this  section 
are  the  meaning  of  “allowable  value”  with  regard  to  representation  of  fixed-point  types,  the 
value  for  'SIZE  of  a  floating-point  type  implemented  on  unusual  hardware,  and  the  accept¬ 
able  consequences  of  a  bad  floating-point-divide  machine  operation.  Appendix  A  contains 
a  more  detailed  discussion  of  these  disputes. 

The  first  dispute  concerned  whether  a  compiler  could  select  a  value  for  'LAST  for 
a  fixed-point  type  and  then  reject  a  'SIZE  representation  specification  for  that  type  that  did 
not  provide  sufficient  space  to  store  that  value,  or  must  'LAST  be  selected  in  accordance 
with  the  specified  representation  size.  The  Fast-Reaction  Team  (FRT)  was  generally 
opposed  to  the  implementation  and  supported  the  tests  but  there  was  some  dissent  from  a 
strong  position  against  the  petitioner.  IDA  analyzed  the  tests  and  noticed  that  they  did  not 
correctly  test  for  the  values  'FIRST  and  'LAST.  In  consideration  of  this  incorrectness  of  the 
tests,  the  divided  opinion  of  the  FRT,  and  the  unclear  wording  of  the  Standard  and  Com¬ 
mentaries,  the  dispute  was  accepted.  The  implementer  was  cautioned  that  the  tests  were  still 
considered  to  reflect  the  intent  of  the  language  and  that  the  revised  language  (Ada  9X) 
would  make  a  similar  requirement. 

The  dispute  concerning  the  appropriate  value  for  'SIZE  for  a  floating-point  type 
arose  as  a  result  of  an  unusual  architecture  that  restricted  access  to  the  bits  that  held  a  float¬ 
ing-point  value's  exponent.  This  hardware  allowed  access  to  only  24  of  32  bits  in  each  word 
for  most  operations;  only  floating-point  operations  use  the  full  32  bits.  The  FRT  again 
reached  no  consensus.  The  dispute  was  accepted  as  the  grounds  for  rejection  were  dubious 
and  the  consequences  of  this  implementation's  behavior  were  unlikely  to  affect  users 
adversely. 

The  third  dispute  explored  the  extent  to  which  a  bad  machine  floating-point  divide 
operation  could  cause  non-conforming  behavior.  The  FRT  agreed  that  the  implementation 
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of  floating-point  division  in  software  would  be  a  waste  of  effort,  in  that  users  would  not 
want  the  inherent  inefficiency.  But  the  FRT  was  divided  in  whether  to  extend  the  acceptance 
of  non-conforming  behavior  to  the  implementation's  ASCII-to-float  conversion  routine, 
which  itself  used  the  faulty  division  operation;  two  FRT  members  argued  that  the  imple- 
menter  should  work  around  the  hardware  problem  just  as  users  must  do  in  applications. 
IDA  accepted  the  non-conforming  behavior  on  floating-point  division  as  justified  as  per 
Commentary  AI-00325  (“it  is  impossible  or  impractical  to  remove  it,  given  the  implemen¬ 
tation's  execution  environment”);  but  IDA  rejected  the  petition  against  the  required  ASCII- 
to-float  conversion  and  required  the  ACVC  test  to  be  passed. 


5.  OTHER  VALIDATION  ACTIVITIES 


One  implementer  requested  that  groups  of  ACVC  test  programs  be  combined  into 
a  single  program  so  that  compilation,  linking,  and  downloading  time  would  be  reduced. 
IDA  worked  with  the  AVF  to  implement  this  special  processing,  which  required  that  indi¬ 
vidual  test  programs  be  modified  so  that  they  could  be  included  as  subprograms  within  a 
enclosing  program  shell.  IDA  analyzed  the  ACVC  in  order  to  determine  what  test  programs 
could  be  so  modified  without  compromising  any  test  objective.  The  effort  was  undertaken 
in  part  for  the  sake  of  assessing  the  benefit  of  such  processing  for  validation  in  general; 
however,  the  AVF's  essential  job  of  collecting  and  grading  results  under  some  pressure  to 
complete  validation  quickly  precluded  collecting  comparative  data  on  the  effects  of  pro¬ 
cessing.  Informal  observations  by  the  AVF  and  some  of  the  implementer  personnel  led  to 
the  conclusion  that  the  special  processing  was  more  trouble  than  benefit,  at  least  when 
implemented  on  an  ad  hoc  basis.  IDA  suggested  to  the  Ada  9X  ACVC  Team  that  the  pro¬ 
cess  be  considered  when  designing  the  Ada  9X  test  suite  (i.e.,  that  there  be  a  means  to  effi¬ 
ciently  implement  this  test-group  processing). 

IDA  also  responded  to  an  implementer's  query  about  a  particular  language  rule 
which  a  few  ACVC  tests  checked.  In  certain  contexts  with  an  access  object  A  that  accesses 
an  array  type,  “A(l)”  is  illegal  while  the  semantically  equivalent  “A.ALL(l)”  is  legal.  The 
query  was  discussed  with  the  FRT  and  the  leader  of  the  Ada  9X  Mapping/Revision  Team 
(MRT).  The  MRT  is  planning  to  revise  the  language  definition  to  remove  the  peculiar  case 
of  legality  that  the  ACVC  tests  check. 


6.  CONCLUSION 


During  this  fiscal  year,  policy  issues  were  effectively  resolved  through  a  consensus 
building  process  with  compiler  vendors,  DoD,  and  non-DoD  organizations.  Not  all  mem¬ 
bers  of  the  Ada  community  supported  these  changes  but  the  objective  of  reducing  the  cost 
for  validation  during  the  period  of  transition  to  Ada  9X  had  been  addressed.  These  costs 
will  not  disappear  but  will  be  incurred  when  a  vendor  responds  to  a  Request  for  Proposal 
(RFP)  for  new  business.  DoD  procurement  officers  are  now  requiring  that  the  compiler  ver¬ 
sion  bid  be  tested  in  the  exact  computer  system  that  is  also  bid.  Conformity  testing  before 
contract  award  is  becoming  more  common  as  vendors  offer  products  that  claim  to  comply 
with  Federal  Information  Processing  Standards  (FIPS). 


APPENDIX  A. 

SUMMARY  OF  FISCAL  YEAR  1992  DISPUTED  TESTS 


A.1  CD2A5IE,  CD2A52A/E/F/I/J,  CD2A54A/E/I/J  (10  Tests) 

These  tests  check  that  operations  on  a  fixed  point  type  are  correct  when  the  type  is 
given  the  smallest  possible  size  with  a  representation  clause.  One  implementation  rejected 
the  length  clause  because  the  specified  upper  bound  of  the  fixed-point  type  had  been  chosen 
as  a  “value  of  the  type”  and  would  not  be  able  to  be  represented  were  the  length  clause 
obeyed.  IDA  accepted  the  petition  against  these  tests  on  the  basis  of  conflicting  wording  of 
the  Ada  standard  and  Commentaries,  and  also  because  the  tests  incorrectly  test  for  the 
attributes  ‘FIRST  and  ‘LAST.IDA  cautioned  the  implementer  to  expect  that  these  tests 
would  likely  be  supported  by  the  language  revision  (Ada  9X).  (However,  there  has  been  no 
validation  of  an  implementation  exhibiting  the  behavior  of  this  implementation.) 

A.2  CD2A81A/B/E,  CD2A83A/B/C/E  (7  Tests) 

These  tests  check  that  operations  on  an  access  type  are  correct  when  the  type’s  size 
is  specified  with  a  representation  clause.  The  standard  customization  of  the  ACVC  allows 
only  one  size  (“$ACC_SIZE”)  for  access  types.  Some  implementations  use  a  larger  size  for 
access  types  whose  designated  type  is  STRING.  The  tests  were  modified  by  incrementing 
the  macro-inserted  specified  size  by  a  constant. 

A.3  C34009D  (1  Test) 

This  test  checks  operations  on  a  derived  record  type  with  an  array  component;  one 
check  is  that  the  record  type’s  ‘BASE’SIZE  is  greater  than  the  sum  of  all  of  the 
components’  ‘SIZEs.  Two  petitioners  have  challenged  this  check  for  an  implementation 
that  includes  only  a  pointer  into  the  heap  for  the  array  component.  Petitions  on  these  points 
are  accepted  because  ISO  WG-9  is  still  deliberating  the  issue  as  Commentary  AI-00825. 
However,  one  petitioner  submitted  an  acceptable  rationale  with  unacceptable 
implementation  results.  Many  of  the  test’s  checks  were  failed  in  addition  to  the  one  that  is 
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affected  by  AI-00825.  This  unsubstantiated  petition  was  rejected,  and  later  implementation 
results  matched  the  previous,  accepted  petitions;  the  test  was  then  allowed  to  be  graded 
PASSED  by  Evaluation  Modification  (of  ignoring  the  one  failed  check). 

A.4  C34003A  (1  Test) 

This  test  checks  that  predefined  operations  are  correctly  implemented  for  derived 
floating-point  types;  it  includes  a  check  on  the  ‘SIZE  attribute.  One  implementation  set 
FLOAT’ SIZE  to  be  24,  even  though  floating-point  values  in  fact  occupy  32  bits.  The  size 
of  24  was  chosen  because  only  floating-point  operations  could  access  one  of  the  4  bytes  of 
a  word.  Essentially,  the  implementer  was  adapting  an  existing  compiler  to  different 
hardware  architecture,  and  considered  the  cost  of  re-implementing  storage  allocation  in  the 
compiler  to  be  excessive.  There  was  support  for  the  petition  on  an  AI-00325  basis,  and  the 
petition  was  accepted:  the  test  was  graded  PASSED  by  Evaluation  Modification  (of 
ignoring  the  one  failed  check). 

A.5  C35902A,  CE3804J,  CE3810B  (3  Tests) 

These  tests  make  various  checks,  some  of  which  depend  on  a  implementation 
supporting  fixed-point  types  of  32  bits.  One  implementation’s  fixed-point  base  type  was  of 
only  24  bits,  where  this  was  the  largest  integer  size.  For  this  implementation,  C359032A 
was  graded  PASSED  by  a  Test  Modification:  one  constant  was  reduced  in  value  to  fit  in  24 
bits.  CE3804J  &  CE3810B  were  graded  INAPPLICABLE  by  Evaluation  Modification 
(accepting  the  compiler’s  rejection  of  the  too-large  fixed-point  type  declarations). 

A.6  C4552IA-E  (5  Tests) 

These  tests  check  floating-point  operations.  One  implementation  for  the  MIL-STD- 
1750A  used  hardware  that  had  a  bad  floating-point  division  operation,  which  affected 
results  on  these  tests.  The  FRT  agreed  that  this  implementer  should  not  compensate  for  the 
bad  machine  instruction  with  a  software  implementation.  These  tests  were  graded  PASSED 
by  Evaluation  Modification  (of  ignoring  the  failed  checks  on  division). 

A.7  CE3809A  (1  Test) 

This  test  checks  that  FLOAT_IO.GET  operates  correctly  from  strings.  One 
petitioner  argued  that  a  bad  machine  floating-point  division  operation  caused  the 


implementation’s  conversion  routing  to  fail  an  accuracy  check.  The  FRT  was  split  on 
whether  the  implementation  should  compensate  for  the  bad  operation,  but  IDA  rejected  the 
petition.  IDA  favored  the  argument  that  the  implementation’s  conversion  routine  should 
compensate  for  the  bad  hardware,  just  as  user  applications  would  have  to  do;  it  was  not 
necessary  that  a  routine  be  compromised  by  the  bad  hardware. 

A.8  EE3301B  (1  Test) 

The  IDA  questioned  this  late  petition  as  being  incomplete,  and  subsequently  the 
petitioner  decided  not  to  pursue  the  issue.  IDA  questioned  the  petition  principally  because 
only  EE301B  was  cited,  but  IDA  believed  that  EE3405B  and  EE3410F  should  also  be 
affected.  If  these  tests  were  not  affected,  further  explanation  of  the  petition  seems 
necessary.  In  each  of  the  three  tests,  test  output  that  was  written  to  STANDARD_OUTPUT 
is  required  to  exhibit  page  breaks  by  explicit  NEW_PAGE  invocations  in  the  unchallenged 
tests,  and  by  output  in  excess  of  PAGE_LENGTH  in  EE3301B. 

A.9  CE3801A/B  (2  Tests) 

These  tests  check  that  STATU  S_ERROR  is  raised  when  TEXTJO  operations  are 
attempted  on  unopened  files;  in  separate  blocks,  calls  are  made  to  GET  and  PUT,  each  with 
the  same  actual  parameter  X.  One  petitioner  argued  that  after  the  failed  call  to  GET 
(STATU S_ERROR  being  raised,  as  expected),  the  parameter  X  was  undefined  because 
optimization  had  eliminated  the  previous  assignment  to  X.  As  a  consequence  of  the 
implementation’s  leaving  X  undefined,  the  following  call  to  PUT  raised 
CON STRAINT_ERROR  instead  of  STATU S_ERROR,  and  REPORT. FAILED  is  called. 
The  petitioner  cited  passages  from  the  Ada  standard  to  support  the  assertion  that  X  may  (or 
even  should)  be  undefined.  The  FRT  generally  did  not  find  the  petition  compelling,  but 
there  was  some  sentiment  that  the  behavior  could  be  allowed  in  consideration  of  the  lack 
of  any  good  guidelines  on  what  effects  optimization  may  have.  IDA  ruled  that  the  tests  may 
be  passed  by  the  following  Test,  Processing,  and  Evaluation  modifications:  insert 
intervening  assignments  to  X  (between  the  GET  and  PUT  operations)  and  process  with  full 
optimization;  process  the  unmodified  tests  without  optimization  (and  obtain  normal 
PASSED  results);  and  process  the  unmodified  tests  with  optimization  (to  witness  the 
behavior  described  by  the  petition).  In  fact,  the  petitioner  changed  the  implementation  and 
passed  the  tests  without  any  modification. 
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